Image processing method, image processing apparatus, and storage medium that extract first information and second information from acquired image data and that process the first information on the basis of the second

ABSTRACT

An image processing method includes an acquisition step of acquiring image data by capturing printed matter, wherein first information and second information, indicating a type of the first information, are embedded in the printed matter as an electronic watermark, an extraction step of extracting the first information and the second information from the image data acquired at the acquisition step, and a processing step of processing the extracted first information by different processing methods in accordance with the extracted second information. Based on that the type of the first information indicated by the second information is a predetermined type, the extracted first information is processed at the processing step, by a predetermined processing method for access to an external device using the extracted first information and display of a web page based on the access.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of copending U.S. patent applicationSer. No. 16/004,697, filed Jun. 11, 2018, which claims the benefit ofJapanese Patent Application No. 2017-126745, filed Jun. 28, 2017, eachof which is hereby incorporated by reference in its entirety.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a technique to embed information in animage and to read the embedded information.

Description of the Related Art

Japanese Patent Laid-Open No. 2013-183255 has disclosed reproducingmoving image data by outputting printed matter in which moving imagedata and information about a storage location of associated informationare embedded in an image and by accessing the storage location based onthe information extracted from captured image data obtained by capturingthe printed matter.

SUMMARY OF THE INVENTION

However, Japanese Patent Laid-Open No. 2013-183255 has not disclosedswitching displays in accordance with a type of additional informationembedded in an image. Because of this, there is a possibility that it isnot possible to appropriately produce a display based on additionalinformation embedded in an image.

Consequently, in view of the above-described problem, an object of thepresent invention is to produce a display in accordance with a type ofadditional information extracted from captured image data.

In one aspect, the present invention is directed to an image processingmethod including a selection step of selecting a type of firstprocessing-target information that is embedded in a printing-targetimage in accordance with instructions by a user, an embedment step ofembedding the first information and second information indicating a typeselected at the selection step in the printing-target image as anelectronic watermark, a printing control step of causing a printingapparatus to print the printing-target image in which the firstinformation and the second information are embedded at the embedmentstep, an acquisition step of acquiring captured image data obtained bycapturing printed matter obtained by printing at the printing controlstep, an extraction step of extracting the first information and thesecond information based on captured image data acquired at theacquisition step, and a processing step of processing the firstprocessing-target information extracted at the extraction step bydifferent processing methods in accordance with the second information.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A and FIG. 1B are block diagrams showing configurations of amobile terminal;

FIG. 2 is a diagram showing a home screen;

FIG. 3A to FIG. 3F are diagrams showing UI screens of a messagecreation/display application;

FIG. 4 is a flowchart of processing to create print data and to printthe print data; and

FIG. 5 is a flowchart of processing to extract additional informationfrom printed matter and to display the additional information.

DESCRIPTION OF THE EMBODIMENTS

In the following, with reference to the attached drawings, preferredembodiments of the present invention are explained in detail. Thefollowing embodiments are not intended to limit the present inventionaccording to the scope of the claims. Further, all combinations offeatures explained below are not necessarily indispensable to thesolution of the problem of the present invention.

First Embodiment

In the present embodiment, print data is created by embedding (alsocalled embedment processing or multiplexing processing) additionalinformation (also called multiplexed information or embeddedinformation) in an image. Then, additional information is taken out fromcaptured image data obtained by capturing printed matter of the printdata (also called extraction processing or separation processing) andthe additional information is displayed in an appropriate format.

The embedment processing, the extraction processing, and the displayprocessing described above are performed in an image processingapparatus (information processing apparatus). It may also be possible touse any apparatus capable of the processing described previously as theimage processing apparatus. However, in the following, for convenienceof explanation, an explanation is given by taking the case as an examplewhere the image processing apparatus is a mobile terminal. As a mobileterminal, mention is made of a personal information terminal, such asPDA (Personal Digital Assistant), a mobile telephone, a smartphone, andso on. A smartphone is a multifunction mobile telephone mounting acamera function, an Internet browser function, a mail function, and soon, in addition to the mobile telephone function.

<About Configuration of Mobile Terminal>

In the following, a configuration of a mobile terminal in the presentembodiment is explained. First, a hardware configuration of a mobileterminal is explained by using FIG. 1A. As shown in FIG. 1A, a mobileterminal 100 includes a CPU 101, a memory 102, a storage unit 103, anoperation unit 104, a display unit 105, a network interface (interfaceis abbreviated to IF) 106, and a camera module 107. These components areconnected via a bus 110 and each is capable of performing transmissionand reception of data with one another.

The CPU 101 is a unit configured to centrally control the mobileterminal 100 and to perform each piece of processing by loading programsstored in the storage unit 103 onto the memory 102 and executing theloaded programs. The memory 102 is, for example, a random access memory(RAM), used as a work area of the CPU 101, and temporarily storesvarious kinds of data. The storage unit 103 is, for example, a flashmemory and stores various programs for performing the processing of themobile terminal 100 and data, such as parameters, necessary for theprocessing of the mobile terminal 100. The programs stored in thestorage unit 103 include various control programs, such as an operatingsystem (hereafter, abbreviated to OS) and applications, and programs forperforming processing to be described later are also stored in thestorage unit 103.

The operation unit 104 is a user interface that receives an operation ofa user and includes buttons, a touch panel, and so on. The display unit105 is a user interface that displays various kinds of information to auser and includes a liquid crystal panel, and the like. Although detailswill be described later, on the display unit 105, a user interfacescreen (hereafter, abbreviated to a UI screen) of an applicationinstalled in the mobile terminal 100 is displayed. The network IF 106 isan interface that connects the mobile terminal 100 to a network. Themobile terminal 100 is capable of connecting to the Internet or externaldevices, such as a printer and a scanner, via the network 106. Thecamera module 107 is a module used for image capturing and it ispossible for a user to operate the camera module 107 through a cameraapplication, and the like, installed in the mobile terminal 100. The CPU101 creates image data based on information received from the cameramodule 107. The created image data is stored in the memory 102 or thestorage unit 103 and displayed on the display unit 105 as needed.

Following the above, a software configuration of the mobile terminal isexplained by using FIG. 1B. As shown in FIG. 1B, the mobile terminal 100includes an OS 120, an application 121, and an application 122. Theseprograms of software are stored in the storage unit 103 and loaded ontothe memory 102 and executed by the CPU 101.

The plurality of applications included in the mobile terminal 100 aredownloaded via a network, such as the Internet, and installed, and it ispossible for a user to make use of each of the plurality of applicationson the mobile terminal 100. The number of applications that the mobileterminal 100 has is not limited to two and the number may be one orthree or more.

The application 121 and the application 122 operate via the OS 120. Forexample, it is possible for these applications to acquire informationacquired by the camera module 107 as image data via the OS 120. Further,for each of the application 121 and the application 122, information(activation information) for activating these applications is defined.For example, in the case when the application 121 transmits activationinformation for activating the application 122, which is created in aformat determined in advance, to the OS 120, the application 122 isactivated by the OS 120. In the present embodiment, for example, theapplication 121 is a message creation/display application and theapplication 122 is a Web browser application or a map application.Details of the message creation/display application will be describedlater.

Further, it is possible for the CPU 101 to create data that is aprinting target by a printer by executing the application 121 and the OS120, and to cause a printer connected to the network IF 106 to print animage based on the data.

<About Display Screen of Mobile Terminal>

In the following, a display screen of the mobile terminal 100 in thepresent embodiment is explained by using FIG. 2 and FIG. 3A to FIG. 3F.FIG. 2 is a diagram showing a home screen that is displayed on thedisplay unit 105 of the mobile terminal 100, specifically, on a liquidcrystal display 201. The home screen is a screen that is the start pointof all the operations and a screen on which icons of applicationsinstalled in the mobile terminal 100 are displayed in a list. In theexample in FIG. 2, six icons, that is, a Phone icon 202, a Photos icon203, a Print icon 204, a Messages icon 205, a Web icon 206, and a Scanicon 207 are displayed. These icons are pressed down (tapped) by a userat the time of activating applications.

In the case when the screen is tapped by a user, the tap of a user isdetected by a touch panel 208 provided on the liquid crystal display201. The touch panel 208 is one of the components of the operation unit104 of the mobile terminal 100 and detects the operation of a user,specifically, detects a tap, a flick, a swipe, and so on.

In the case when a user taps the icon of an application, the tap of theuser is detected by the touch panel 208 and an event corresponding tothe tapped icon, that is, a graphical user interface (GUI) part (alsocalled an object) is issued. As a result of this, the applicationcorresponding to the touched icon activates. For example, in the casewhen the Web icon 206 is tapped in the state shown in FIG. 2, a Webbrowser activates.

Further, the mobile terminal 100 includes a home button 209. By pressingdown the home button 209 while activating the application, it ispossible to close the screen of the application being activated and toreturn to the home screen.

FIG. 3A is a diagram showing a UI screen displayed on the liquid crystaldisplay 201 in the case when the Messages icon 205 in FIG. 2 is tappedand the message creation/display application activates. The messagecreation/display application is an application that performs processingto create print data in which data (meta data) relating to a message isembedded in an image and processing to display a message based oncaptured image data obtained by capturing printed matter of the printdata.

The UI screen shown in FIG. 3A has three objects for a user to select atype of a message to create. That is, in the present embodiment, at thetime of creating a decoration message, an object 311 is tapped.Similarly, at the time of creating a moving image message, an object 312is tapped and at the time of creating a text/Web message, an object 313is tapped. In the present embodiment, the message creation/displayapplication embeds one of the three kinds of message data in an image inaccordance with a selection of a user, but the number of types ofmessage data that the message creation/display application can create isnot limited to three. It may also be possible for the messagecreation/display application to be capable of creating message data of atype other than the three types described previously.

The decoration message is a message of the type in which a message inputby a user is displayed while animation and music prepared in advance foreach theme are being reproduced. The moving image message is a messageof the type that reproduces a moving image prepared by a user anddisplays a message by a moving image. The text/Web message is a messageof the type in which a message input by a user is displayed in the textformat and along with the message, information accompanying this, thatis, the date and time, the position, the link to an arbitrary Web page,and so on, are displayed.

By tapping one of the objects 311 to 313, a UI screen for creating printdata by embedding message data corresponding to the tapped object in animage is displayed. It is possible for a user to create print data inwhich message data is embedded in an image via this UI screen.

FIG. 3B is a diagram showing an example of a UI screen for decorationmessage creation in the message creation/display application, which isdisplayed in the case when a user creates a decoration message bytapping the object 311. An image 321 is an image that a user selects toperform printing by embedding a message. In the present embodiment, inthe case when the image 321 is tapped, it is possible to change an imagethat is displayed on the UI screen for decoration message creation to animage different from the image currently selected. Alternatively, it mayalso be possible to provide a button to select or to change an image onthe UI screen for decoration message creation separately from the image321. Further, it is also possible to select an image of analready-captured photo as a printing-target image that is displayed onthe UI screen for decoration message creation, or to use an image of aphoto newly captured on the spot.

It is possible for a user to edit a message that is displayed in eachscene in animation of a decoration message by tapping a text box 322 ora text box 323. A reproduce button 324, a return button 325, and aforward button 326 are buttons for controlling a preview of a decorationmessage. In the case when a user taps the reproduce button 324, adecoration message is reproduced from the beginning. Further, in thecase when a user taps the return button 325, it is possible to return tothe scene one previous of the scene currently being reproduced and, inthe case when a user taps the forward button 326, it is possible toproceed to the scene one ahead of the scene currently being reproduced.Further, in the case of FIG. 3B, the message “Happy birthday.” input tothe text box 322 is displayed in the first scene and the message“Congratulations!” input to the text box 323 is displayed in the secondscene, and, therefore, it is known that the second scene is beingreproduced.

In the case when the object 312 is tapped in FIG. 3A, an input screenfor inputting information relating to a moving image message isdisplayed. For example, a screen for a user to input a URL at the movingimage site on a Web is displayed. Further, in the case when the object313 is tapped in FIG. 3A, an input screen, on which a message can beinput and information accompanying the message (date, position, URL ofarbitrary Web page, and so on) can be input, is displayed. After theseinput screens are displayed, or after the input screen shown in FIG. 3Bis displayed, a printing instructions screen on which a user can giveinstructions as to printing is displayed on the display unit 105 by themessage creation/display application. Then, in the case wheninstructions as to printing are given on the printing instructionsscreen, the CPU 101 creates data based on the image in which additionalinformation input by a user is embedded and transmits the data to theprinter by the message creation/display application and the OS 120. Theprinter to which the data is transmitted is a printer connected by anetwork I/F and the identification information (MAC address, IP addressin wireless LAN, and so on) for identifying the printer is registered inthe message creation/display application. In the case when a pluralityof printers is registered, in accordance with the identificationinformation on the printer selected by a user, printing-target data istransmitted to the printer.

FIG. 3C is a diagram showing a UI screen that is displayed at the timeof performing image capturing by using the message creation/displayapplication with the mobile terminal 100 being held above printed matter331 of the print data created by using the message creation/displayapplication. The printed matter 331 is printed matter obtained by one ofthe objects 311 to 313 being selected in FIG. 3A as described above andby an image in which additional information is embedded being printed bythe printer as described above after a user inputs the additionalinformation. As shown in FIG. 3C, on the UI screen, an image captured bythe camera module 107, which the message creation/display applicationhas acquired from the OS 120, is displayed.

Further, it is possible to display a message (that is, a decorationmessage, a moving image message, a text/Web message) on an appropriateUI screen by analyzing message data embedded in an acquired image byusing the message creation/display application.

FIG. 3D to FIG. 3F are examples of a UI screen representing results ofanalyzing an image acquired from the OS 120 after capturing the printedmatter 331 by using the message creation/display application. FIG. 3Dshows an example of a UI screen for displaying a decoration message,that is, a UI screen for reproducing music and animation, as well asdisplaying a message input by a user. FIG. 3E shows an example of a UIscreen for displaying a moving image message, that is, a UI screen forreproducing a moving image existing at the URL destination included inmessage data.

FIG. 3F shows an example of a UI screen for displaying a text/Webmessage. In this example, the message and the date are displayed andfurther, a button 341 to open a Web page and a button 342 to display amap and clearly specify the position indicated by the positioninformation on the map are displayed. These display items are notdisplayed in the case when data corresponding to each display item isnot included in message data.

It is supposed that additional information is embedded in the printedmatter 331 by a user specifying the object 311 at the time of printingof the printed matter 331. In this case, the UI screen shown in FIG. 3Dis displayed at the time of the printed matter 331 being captured byusing the message creation/display application. Similarly, in the casewhen additional information is embedded in the printed matter 331 by auser specifying the object 312, the UI screen shown in FIG. 3E isdisplayed at the time of the printed matter 331 being captured. Further,in the case when additional information is embedded in the printedmatter 331 by a user specifying the object 313, the UI screen shown inFIG. 3F is displayed at the time of the printed matter 331 beingcaptured.

The message creation/display application embeds informationcorresponding individually to the objects 311 to 313, respectively, inthe printed matter 331 at the time of printing of the printed matter331. That is, the information capable of specifying the type of themessage to be embedded is embedded along with the contents of themessage. Then, the message creation/display application analyzes themessage data embedded in the image acquired from the OS 120 and displaysthe message by using the UI screen in accordance with the type of themessage (message type). This will be described later by using FIG. 5.

Here, a case is explained when the processing to embed additionalinformation and the processing to extract additional information areperformed by one application. That is, the same message creation/displayapplication performs both the processing to create print data in whichmessage data is embedded in an image and the processing to extractmessage data from captured image data obtained by capturing printedmatter and to display a message. However, the present embodiment is notlimited to this case. For example, as another embodiment, an aspect isconsidered in which the processing to create print data in which messagedata is embedded in an image and the processing to extract message datafrom captured image data obtained by capturing printed matter and todisplay a message are performed by different applications, respectively.In this case, the format of message data used by both the applicationsis determined in advance and the application that creates print datacreates print data by creating message data in that format and embeddingthe message data in an image. On the other hand, the application thatdisplays a message analyzes message data in the format determined inadvance and displays a message on an appropriate UI screen.

<About Processing to Create Print Data in which Additional Informationis Embedded in Image and to Print the Print Data>

In the following, the processing to create print data in whichadditional information, that is, message data is embedded in an image byusing the message creation/display application and to print the printdata in the present embodiment is explained by using FIG. 4. Thefollowing processing at each step is performed by the CPU 101. Further,the processing shown in FIG. 4 is performed, for example, in the casewhen instructions as to printing are given after the message type isselected (one of the objects 311 to 313 is tapped) and additionalinformation is input by a user.

At step S401, the CPU 101 functions as a format determination unit anddetermines the format of the creation-target message data in accordancewith the message type selected by a user. In the present embodiment, foreach piece of message data of a decoration message, a moving imagemessage, and a text/Web message that can be created by using the messagecreation/display application, a corresponding format is determined inadvance. That is, specifically, at this step, the format of message datais determined in accordance with which of the object 311, the object312, and the object 313 is tapped on the UI screen shown in FIG. 3A.

In the case when the format of message data is determined uniquely atstep S401, a UI screen (for example, FIG. 3B, and the like) for creatingmessage data in the determined format is displayed on the messagecreation/display application. It is possible for a user to select imagedata that a user desires to print, such a photo, to input a message thatis added to the selected image data, and so on, via this UI screen.Further, it is also possible for a user to specify a color and a font ofthe character for the input message via this UI screen.

In the case when a user input via the UI screen after step S401 iscompleted, at one of steps S402 to S404, information that is embedded ina printing-target image, that is, message data in the formatcorresponding to the selected message type is created. That is, in thecase when the format is determined to be the format of a decorationmessage at step S401, the CPU 101 functions as a message data creationunit at step S402 and creates message data in the format of a decorationmessage. Alternatively, in the case when the format is determined to bethe format of a moving image at step S401, message data is created inthe format of a moving image at step S403. Alternatively, in the casewhen the format is determined to be the format of a text/Web message atstep S401, message data is created in the format of a text/Web messageat step S404.

For message data that is created in the present embodiment, the formatthereof is defined for each message type. That is, the message datacreated at steps S402 to S404 is different in format from one anotherand, in the message data, information in accordance with each messagetype is included.

For example, in message data created in the decoration message-specificformat, an identifier indicating that the message is a decorationmessage and information on (parameter of) a theme specifying animationand music to be reproduced are included. Here, as the theme,specifically, mention is made of a birthday, Christmas, and so on.Further, in message data in the format of a decoration message, contentsof a message (that is, character string) input by a user are included,and it is also possible to cause the message data to include informationon the character, color, and font at the time of displaying the message,and the like. In the following, an explanation is given by using aspecific example “ACEBADXX¥EAFHappy birthday. Congratulations!¥TCBHave anice day!” of message data created in the decoration message-specificformat.

In the above-described message data, A is a parameter of an identifierof a message type indicating that the message is a decoration messageand CE is a parameter of a theme ID specifying a theme (in this example,birthday). BA that follows is a parameter specifying the font of themessage, D is a parameter specifying the character color of the message,and “XX” is the contents of the message. ¥ that follows is a separatorseparating the message, EA is a parameter specifying a font of themessage, F is a parameter specifying the character color of the message,and “Happy birthday. Congratulations!” is the contents of the message. ¥that follows is a separator, TC is a parameter specifying the font ofthe message, B is a parameter specifying the character color of themessage, and “Have a nice day!” is the contents of the message.

In message data created in the moving image message-specific format, anidentifier indicating that the message is a moving image message and aURL of the moving image to be reproduced are included. In the following,an explanation is given by using a specific example“Bhttp://www.abcde.com/abc” of message data created in the moving imagemessage-specific format.

In the above-described message data, B is a parameter of an identifierof a message type indicating that the message is a moving image messageand the character string that follows is a URL of the moving image.

In message data created in the text/Web message-specific format, anidentifier indicating that the message is a text/Web message andcontents (that is, a character string) of the message input by a userare included. Further, it is also possible to cause the message data inthis format to include information on a character color and a font atthe time of displaying the message, date information, positioninformation, a URL of a Web page, and so on. Here, the date informationis, for example, information indicating the date, such as the day onwhich the text/Web message was created, and the position information is,for example, information indicating the position, such as the positionat which the photo was captured. In the following, explanation is givenby using a specific example “CABADWe had a goodtime!¥Bhttp://www.abcde.com¥CBAD20160719¥D129374000967” of message datacreated in the text/Web message-specific format.

In the above-described message data, C is a parameter of an identifierof a message type indicating that the message is a text/Web message andA is a parameter indicating that the character string that follows isthe message input by a user. BA that follows is a parameter specifyingthe font of the message, D is a parameter specifying the character colorof the message, and “We had a good time!” is the contents of themessage. ¥ that follows is a separator, B is a parameter indicating thatthe character string that follows is the URL of a Web page, and“http://www.abcde.com” is the URL. ¥ that follows is a separator, C is aparameter indicating that the character string that follows is dateinformation, BA is a parameter specifying the font at the time ofdisplaying the date indicated by date information, and D is a parameterspecifying the character color at the time of displaying the date.Further, “20160719” that follows is the date and, in this example, thedate indicates Jul. 19, 2016. ¥ that follows is a separator, D is aparameter indicating that the character string that follows is positioninformation, and “129374000967” is the contents of position information.As described above, in this example, the position information isincluded in the message data as information on the latitude and thelongitude.

Here, the three kinds of message types are explained. but the messagetype is not limited to the three kinds of message type. Further,information included in the message data of each message type is notlimited to that described above, and any other pieces of information maybe included. For example, in the case when an arbitrary stamp can beadded to an image on the UI screen of the message creation/displayapplication, it is possible for the message data to include informationon the stamp. On the other hand, it is not necessary for the messagedata to include all the pieces of information determined as the messageformat and it is possible for a user to selectively input only theinformation the user desires to input.

At step S405, the CPU 101 functions as a print data creation unit andcreates print data by embedding message data created at one of stepsS402 to S404 by a method that is difficult for a user to determine (forexample, an electronic watermark, and the like). The method of embeddingmessage data in an image is not limited to a method that is difficultfor a user to determine, such as an electronic watermark. The messagecreation/display application may use an arbitrary method by whichembedded message data can be determined, and, for example, a method thata user can determine, such as a barcode and a two-dimensional code.

At step S406, the CPU 101 performs printing processing to transmit aprint job including the print data created at step S405 to the externalimage forming apparatus in accordance with printing instructions of auser via the UI screen of the message creation/display application. Dueto this, the printed matter in which additional information, that is,message data is embedded is output. In the case when message data isembedded by a method that is difficult for a user to determine, forexample, the additional information is embedded for each unit block(multiplexed block) across the surface of the printed matter. The shapeof the area where information to be used is embedded is not limited to arectangle, and it may also be possible to use a unit area in any shapeas an area where information is embedded. The above is the contents ofthe processing to create print data in which additional information isembedded in an image and to perform printing in the present embodiment.

<About Processing to Extract Additional Information from Printed Matterand to Display Extracted Additional Information>

In the following, processing to extract additional information, that is,message data from printed matter, to analyze the extracted message data,and to display the message by using the message creation/displayapplication in the present embodiment is explained by using FIG. 5. Thefollowing processing is started in the case when the mobile terminal 100is held above printed matter output by the printing processing at stepS406 and the printed matter is captured by using the messagecreation/display application. Further, the processing at each step isperformed by the CPU 101.

At step S501, captured image data is acquired. The captured image dataacquired at this step may be still image data or moving image data.

At step S502, from the captured image data acquired at step S501,additional information, that is, message data is extracted. As describedabove, this message data is data embedded in an image by the messagecreation/display application. There is a case when the message data isembedded by a method that is difficult for a user to determine, such asan electronic watermark, or a case when the message data is embedded bya method that a user can determine, such as a barcode and atwo-dimensional code.

At step S503, the CPU 101 functions as a message type determination unitand determines the message type based on the message data extracted atstep S502. Specifically, based on the identifier of the message typeincluded in the extracted message data, whether the message type is adecoration message, a moving image message, or a text/Web message isdetermined. That is, in the case when the identifier indicating adecoration message is included in the extracted message data, themessage type is determined to be a decoration message. Similarly, in thecase when the identifier indicating a moving image message is includedin the extracted message data, the message type is determined to be amoving image message and in the case when the identifier indicating atext/Web message is included, the message type is determined to be atext/Web message. The determination processing at this step is importantprocessing for switching UI screens to be displayed after this step. Dueto the determination processing at this step, it is made possible toproduce a display suitable to each message type at subsequent steps. Inthe case when the processing at step S503 is not performed, it is nolonger possible to appropriately display the message of each type in thesubsequent processing. For example, such a case may occur when, in thedisplay of a decoration message, neither animation nor music isreproduced, and only a message is displayed, or when in the display of atext/Web message, the character string of a message is displayed as aWeb link.

In the case when it is determined that the message type is a decorationmessage at step S503, a UI screen for a decoration message is displayedat step S504 and, on the UI screen for a decoration message, an imageand a decoration message are displayed. Specifically, animation andmusic corresponding to information (parameters) of the theme included inthe message data are reproduced on the UI screen for a decorationmessage. Further, in the case when a message input by a user is includedin the message data, the message is displayed on the UI screen for adecoration message in harmony with the animation and music. In the casewhen parameters specifying the character color, the font, and so on, ofthe message are included in the message data, in addition to the messageinput by a user, the message is displayed by using the specifiedcharacter color and font.

In the case when it is determined that the message type is a movingimage message at step S503, a UI screen for a moving image message isdisplayed at step S505 and, on the UI screen for a moving image message,a moving image existing at the URL included in the message data isreproduced. The UI screen, on which a moving image is reproduced, is notlimited to the UI screen of the message creation/display application. Itmay also be possible to deliver the URL to another application, such asa Web browser and a moving image browser application, and to reproduce amoving image on the UI screen of the application other than the messagecreation/display application. For example, the application 121 as amessage creation/display application transmits the activation command ofthe application 122 as a Web browser to the OS 120 along with the URL.Then, in the case when the OS 120 activates the Web browser and deliversthe URL to the Web browser, the activated Web browser displays a Webpage in accordance with the URL. Further, in such a case, it may also bepossible to automatically perform control to determine an application onthe UI screen of which a moving image is reproduced on the messagecreation/display application. Alternatively, it may also be possible toenable a user to select which of the message creation/displayapplication and another application to use to reproduce a moving imageby presenting alternatives on the UI screen of the messagecreation/display application.

In the case when it is determined that the message type is a text/Webmessage at step S503, a UI screen for a text/Web message is displayed atstep S506. Then, on the UI screen for a text/Web message, a message, adate, and so on, are displayed based on the message data. At this time,in the case when parameters specifying the character color and the fontare included in the message data, a message and a date are displayed byusing the specified character color and font. Further, in the case of atext/Web message, there is a case when position information is includedin the message data. In this case, it may also be possible to display amap on the UI screen of the message creation/display application and toset a pin at the coordinates indicated by the position information onthe displayed map. Alternatively, it may also be possible to activate amap application separate from the message creation/display application,to display a map on the UI screen of the map application, and to set apin on the displayed map. The activation method of a map application andthe delivery method of position information are the same as theactivation method of a Web browser and the delivery method of a URL,respectively, described above. Further, in the case of a text/Webmessage, there is a case when the URL of a Web page is included in themessage data. In this case, as in the case of a moving image messagedescribed above, it may also be possible to display a Web page on the UIscreen of the message creation/display application, or to display a Webpage by an application other than the message creation/displayapplication, for example, by a Web browser. In the case when it ispossible to include a plurality of pieces of information in one piece ofmessage data like a text/Web message, only the information extractedfrom the message data is displayed on the UI screen for a text/Webmessage. As an example, a case is explained where the contents of themessage, the date information, and the URL of a Web page are included inthe message data and position information is not included. In this case,on the UI screen for a text/Web message, the item relating to positioninformation is not displayed and the message, the date, and the link toa Web page are displayed. Alternatively, in the case when a plurality ofpieces of information is included, it may also be possible to display abutton, a checkbox, and so on, corresponding to each piece ofinformation first and then to display corresponding information in thecase when the button or the checkbox is operated by a user.

<About Effect, Modification Example, and so on, of the PresentEmbodiment>

As described above, in the present embodiment, it is possible to includeat least one of information, which is a message itself, and informationfor accessing the outside, such as a URL, in message data that isembedded in an image. Due to this, in the case when information, whichis a message itself, is included in the message data, it is possible todisplay the message without network connection and without the need toaccess an external device. On the other hand, in the case wheninformation for accessing an external device, that is, a URL is includedin the message data, the message is displayed by accessing the externaldevice. As described above, it is not necessarily required to connect toa network and to access an external device to display a message and itis only required to connect to a network only in the case when it isnecessary in accordance with analysis results of the extracted messagedata.

Further, in the case when the results of analyzing the extracted messagedata make it clear that an application other than the messagecreation/display application needs to be used, it may also be possibleto activate another application without an operation of a user. As anexample, a case is considered where only one of the URL of a movingimage, the URL of a Web page, and the position information is includedin the message data, other than the identifier of the message type. Inthis case, by activating a moving image browser application, a Webbrowser, or a map application at the point in time of the analysis ofthe extracted message data, it is possible to reduce a user operationand operability improves.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or an apparatus that reads out and executescomputer executable instructions (e.g., one or more programs) recordedon a storage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., an application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., a central processingunit (CPU), or a micro processing unit (MPU)) and may include a networkof separate computers or separate processors to read out and to executethe computer executable instructions. The computer executableinstructions may be provided to the computer, for example, from anetwork or the storage medium. The storage medium may include, forexample, one or more of a hard disk, a random-access memory (RAM), aread only memory (ROM), a storage of distributed computing systems, anoptical disk (such as a compact disc (CD), a digital versatile disc(DVD), or a Blu-ray Disc (BD)™) a flash memory device, a memory card,and the like.

Due to the present invention, it is made possible to produce a displayin accordance with the type of additional information extracted fromcaptured image data.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

I claim:
 1. An image processing method comprising: an acquisition stepof acquiring image data by capturing printed matter, wherein firstinformation and second information, indicating a type of the firstinformation, are embedded in the printed matter as an electronicwatermark; an extraction step of extracting the first information andthe second information from the image data acquired at the acquisitionstep; and a processing step of processing the extracted firstinformation by different processing methods in accordance with theextracted second information, wherein, based on that the type of thefirst information indicated by the second information is a predeterminedtype, the extracted first information is processed at the processingstep, by a predetermined processing method for access to an externaldevice using the extracted first information and display of a web pagebased on the access.
 2. The image processing method according to claim1, wherein the acquisition step, the extraction step, and the processingstep are performed by a predetermined application.
 3. The imageprocessing method according to claim 2, wherein the extracted firstinformation is processed by the predetermined processing method at theprocessing step, such that the access to the external device and thedisplay of the web page are performed by a web browser different fromthe predetermined application.
 4. The image processing method accordingto claim 3, wherein the extracted first information is processed by thepredetermined processing method at the processing step, such that thatthe web browser is activated and the access to the external device andthe display of the web page are performed by the activated web browser.5. The image processing method according to claim 3, wherein, based onthat the type of the first information indicated by the secondinformation is a second type, different from the predetermined type, theextracted first information is processed at the processing step by thepredetermined application without using the web browser.
 6. The imageprocessing method according to claim 3, wherein, based on that the typeof the first information indicated by the second information is thepredetermined type, the extracted first information is processed at theprocessing step as a uniform resource locator (URL).
 7. The imageprocessing method according to claim 2, wherein, at the processing step,a map application different from the predetermined application is causedto display a map on a display device by the predetermined processing, inaccordance with position information as the first information.
 8. Theimage processing method according to claim 1, wherein, at the processingstep, display processing is performed based on the first information, bydifferent display methods in accordance with the second information, andwherein the different display methods include a display method for thedisplay of the web page.
 9. The image processing method according toclaim 8, wherein, based on that the type of the first informationindicated by the second information is a second type, different from thepredetermined type, the extracted first information is displayed on adisplay device by the display processing at the processing step.
 10. Theimage processing method according to claim 9, wherein, at the processingstep, the first information is displayed as text.
 11. The imageprocessing method according to claim 8, wherein, based on that the typeof the first information indicated by the second information is a secondtype, different from the predetermined type, animation corresponding tothe first information is reproduced in a display device by the displayprocessing.
 12. The image processing method according to claim 8,wherein, based on that the type of the first information indicated bythe second information is a second type, different from thepredetermined type, a predetermined button is displayed on a displaydevice by the display processing, and a display is produced by thedisplay processing, based on the first information, in accordance withan instruction of a user for the predetermined button.
 13. An imageprocessing apparatus comprising: an acquisition unit configured toacquire image data by capturing printed matter, wherein firstinformation and second information, indicating a type of the firstinformation, are embedded in the printed matter as an electronicwatermark; an extraction unit configured to extract the firstinformation and the second information from the image data acquired bythe acquisition unit; and a processing unit configured to process theextracted first information by different processing methods inaccordance with the extracted second information, wherein, based on thatthe type of the first information indicated by the second information isa predetermined type, the extracted first information is processed bythe processing unit, by a predetermined processing method for access toan external devise using the extracted first information and display ofa web page based on the access.
 14. A non-transitory computer readablestorage medium storing a program for causing a computer to perform animage processing method, the image processing method comprising: anacquisition step of acquiring image data by capturing printed matter,wherein first information and second information, indicating a type ofthe first information, are embedded in the printed matter as anelectronic watermark; an extraction step of extracting the firstinformation and the second information from the image data acquired atthe acquisition step; and a processing step of processing the extractedfirst information by different processing methods in accordance with theextracted second information, wherein, based on that the type of thefirst information indicated by the second information is a predeterminedtype, the extracted first information is processed at the processingstep, by a predetermined processing method for access to an externaldevise using the extracted first information and display of a web pagebased on the access.